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Abstract 

In this work we study, analytically and employing Monte Carlo simula- 
tions, the influence of the competition between several activity-dependent 
synaptic processes, such as short-term synaptic facilitation and depres- 
sion, on the maximum memory storage capacity in a neural network. In 
contrast with the case of synaptic depression, which drastically reduces 
the capacity of the network to store and retrieve "static" activity patterns, 
synaptic facilitation enhances the storage capacity in different contexts. 
In particular, we found optimal values of the relevant synaptic parame- 
ters (such as the neurotransmitter release probability or the characteristic 
facilitation time constant) for which the storage capacity can be maxi- 
mal and similar to the one obtained with static synapses, that is, without 
activity-dependent processes. We conclude that depressing synapses with 
a certain level of facilitation allow to recover the good retrieval properties 
of networks with static synapses while maintaining the nonlinear charac- 
teristics of dynamic synapses, convenient for information processing and 
coding. 



1 Introduction and model 

One interesting topic which arises in the study of biologicaUy motivated neural 
systems is how dynamical processes affecting the synapses in different time scales 
can influence the collective behaviour of large neural assemblies. A well known 
example is the slow change in the synaptic strength due to a learning process 
of activity patterns, that can be modelled as a neural network which includes a 
hebbian prescription for the synaptic intensities ( |Hebb, 1949[ ) . As a consequence 
of this process, the network is able to retrieve previously learned patterns, mim- 
icking the associative memory tasks which occur in the brain ( |Amari, 1972[ 
IHopfield, 19"82l |Amit et al., 1987aD . The maximum number of such patterns 



(Afmax), per neuron, that the network is able to store without having signif- 
icant errors in the recovery task is called the critical (or maximum) storage 
capacity of the system. This capacity is commonly denoted by the ratio ac = 
M-axa^/N , with N being the number of neurons in the network. It is well known 
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that the critical storage capacity is affected by certain considerations about 
real neural systems, such as constraints in the range of values of the synaptic 



strength (Fusi and Abbott, 2007), the mean activity level of the stored patterns 
dTsodyks and Feigelman, 1988[ |Amit and Tsodyks, 1991 ), or the topology of 



the network ( McGraw and Menzinger, 2003[ Torres et al., 2004] ). None of these 



works take into account, however, that the synaptic strength also varies in short- 
time scales producing a fluctuating response which can be depressed (synaptic 
depression) or enhanced (synaptic facilitation) depending on the presynaptic ac- 
tivity dAbbott et al., 1997[ [Markram et al., 1998| [Tsodyks et al., 1998P , which 
defines the so called dynamic synapses. 

Short-term depression and facilitation have been proposed as dynamical pro- 
cesses responsible for many kinds of complex behaviour found in neural systems. 
For instance, recent theoretical works have reported the importance of these 
mechanisms in the appearance of periodic and chaotic switching between stored 
activity patterns ( [Pantic et al., 2002| [Torres et al., 2007| , which could be re- 
lated to the oscillations between up and down cortical states ( [Holcman and Tsodyks, 2006] ). 
They are also responsible for the generation of persistent activity in working 



memories (Romani et al., 2006 Barak and Tsodyks, 2007 Mongillo et al., 2008), 



and enhance the detection of correlated inputs in a background of noisy activ- 
ity under different conditions ( [Mejias and Torres, 2008[ ). Synaptic depression 
occurs due to the existence of a limited amount of neurotransmitter vesicles in 
the synaptic button, ready to be released into the synaptic cleft if a presynaptic 
action potential (AP) arrives. This produces, for a high frequency stimulus, 
a decrease of the postsynaptic response — which is a measure of the synaptic 
strength — as is shown in (Abbott et al., 1997). On the other hand, synap- 



tic facilitation takes into account the effect of the influx of extracellular cal- 
cium ions into the neuron near the synapse after the arrival of each presynaptic 
AP ( [Bertram et al., 1996p . These ions bind to some sensor which favours the 
neurotransmitter vesicle depletion, in such a way that the postsynaptic response 
increases for successive APs ( [Kamiya and Zucker, 1994 ). Facilitation, therefore. 



increases the synaptic strength for high frequency presynaptic stimuli. The ef- 
fect of the competition between these two a priory opposite mechanisms has 
been shown to be highly relevant in the emergent behaviour of attractor neural 
networks (ANN) with activity-dependent dynamic synapses with a finite num- 
ber of stored patterns ( Torres et al., 2007| . However, until now the effect of 



such competition in the critical storage capacity has not been reported. Only 
very recently, a few studies have analysed this particularly interesting issue 
for depressing synapses, and showed that the critical storage capacity of sta- 



ble memory patterns is severely reduced in this case (Bibitchkov et al., 2002 



Torres et al., 2002 Matsumoto et al., 2007). Our aim in this work was to com- 



pute the critical storage capacity of an ANN, with both depressing and facil- 
itating mechanisms competing in the synapses, to quantitatively analyse the 
effect of including facilitation in the system. We demonstrate that synaptic 
facilitation improves the storage capacity with respect to the case of depress- 
ing synapses, for a certain range of the synaptic parameters. Moreover, if the 
level of depression is not very large, facilitation can increase the critical stor- 
age capacity, reaching in some cases the value obtained with static synapses 
— which is the maximum that one can obtain considering a hebbian learning 
rule with unbiased random patterns in a fully connected network. Our results 
suggest that a certain level of facilitation in the synapses might be positive for 
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an efficient memory retrieval, while the function of strongly depressed synapses 
could be more oriented to other tasks concerning, for instance, the dynamical 
processing of data. 

Our starting point is a fully connected network of N binary neurons whose 
state s = {si = 0, 1; V« = 1, . . . ,7V} follows a probabilistic Little (parallel and 
synchronous) dynamics ( |Peretto, 1992^ : 

Ph(i+l) = l]-i{l + tanh[2/3(/i,(s,i)-a0]} V^ = 1 . . . TV, (1) 

where hi{s,t) is the local field or the total input synaptic current to neuron j, 
namely 

/i,(s,i) = $^^ya;,(t)F,(i)s,(t). (2) 

Here, /3 — T^^ is a temperature or noise parameter (i.e., for {3 —> oo we 
have a deterministic dynamics), and 9i represents the neuron firing thresh- 
old. The coefficients ujij are fixed synaptic conductances, consequence of the 
slow learning process of M memory patterns of activity. In the following we 
will choose a hebbian prescription for such learning via the standard covariance 
rule dAmit et al., 1987b[ [Tsodyks and Feigelman, 1988D 

1 

-^.-^vnwyBer (3) 

where {^f — 0,1; i = 1 . . . N} represents the M stored random patterns with 
mean activity (^f) — f — 1/2. On the other hand, the variables Xj , Fj appearing 
in hi describe the short-term depression and facilitation synaptic mechanisms, 
respectively. We assume that these variables evolve according to the discrete 
dynamics ( |Tsodyks et al., 1998| [Torres et al, 2002^ 

I — x-(t) 

x,{t + l)^ X, {t) + ■'^ ' - UseF, {t)x, {t)sj it) (4) 

'rec 

u,{t + 1) = u,{t) + ^SE-u,{t) ^ ^^^^^ _ ^^.(^))^^.(^)^ 

Tfac 

where Uj{t) = UsEFjit). Here, Use represents the maximum fraction of neu- 
rotransmitters which can be released in absence of facilitation each time a 
presynaptic AP arrives to the synapse, and Tree, Tfae are, respectively, the time 
constants for depressing and facilitating processes. The dynamics allows 
to recover the critical storage capacity of the standard Hopfield model (ac — 
0.138) ( |Amit et al., 1985] ) for static synapses, that is, when Xi = Fi — 1, V i,t. 
By a simple inspection of Eqs. ([US]), this limit corresponds to the case of 
Tree, Tfae ^ 1 which makes Xj and Uj Vj to quickly reach their maximum val- 
ues, namely 1 and Use, and implies Xj = Fj = I Vj, In this limit one has 
the relation 2[hi{s, t) — 9i] ~ hf {s, t) where hf{s, t) stands for the local field of 



^Note that, more precisely, the static synapse Umit is obtained for TrecTfac ~* Oi but due 
to the discrete dynamics represented by Eqs. H4I5P one can have some dynamical instabilities 
during the simulation of the map for very small time constants. However, a continuous version 
of the dynamics H4I5| I or considering only steady-state conditions (as we assume in this work) 
allows to consider without any problem that limit. 
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the classic Hopfield model with zero threshold, which assumes a { — 1, +1} code 
for the neuron states and implies for 9i the form 

Instead, we used in this work the {1,0} code because it is more related with 
biology and allows for a clear separation of the synaptic current ft.i(s, t) from the 
neuron threshold 9i and, therefore, it enables one to study the effect of synap- 
tic depression and facilitation alone, without including other adaptive effects 
related, for instance, with threshold dynamics. 



2 Mean-field analysis 

From the definition of hi{s, t) and Eqs. ^ and ©, we obtain 

2[h,{s,t) - e,] =^efm^(s,t)--2Qa;,(t)F,(0s,(t) + a (7) 

where « ^ f , ^ 2^ - 1, and m^^{s, t) ^ e^[2x, it)F,{t)s,{t) - 1]. 

We assume now that the system reaches some stationary state (t — > oo) 
which corresponds to a fixed point of the dynamics. In order to work with the 
term Xi{t)Fi{t)si{t) and to obtain an approximately valid mean field theory, we 
also assume that the working temperature (T) in the system is very small (to 
avoid as much as possible thermal fluctuations). This hypothesis is reasonable 
because our goal is to compute maximum storage capacity, so we have to perform 
the limit T ^ 0. One has then two possible scenarios: 

(a) T = 0. The state of the system is quenched and it does not present any 
temporal fluctuations in Si. Therefore, one can assume that in each site 
Si takes a fixed value (namely s,°° = 1,0) for all times. We can then 
evaluate the fixed point in Eqs. (jl]) and (O as a function of s,°° and 
obtain ( [Matsumoto et al., 2007D 

F-= 1 + "/-^?° . = I (8) 

" 1 + Use Tfac 5°° ' * 1 + UseF, Tree 5°° ' 

Taking into account that s°° takes the values {0, 1}, we can simplify the 
expression for the product XiFiSi, leading to 

XiFiSi ~ - ■ -s,j (9) 
1 + 77' 

where 7 = UsETree and 7' = -^^^^J"^"^ — . One can easily check that the 
static limit (Hopfield model) is obtained again for Tree, Tfac which 
implies 7 — > 0, 7' — > 1, respectively. 

(b) T~0(l<c/3<oo). For very low temperatures and in the steady 
state, the typical time interval between thermal fluctuations is very large 
compared with Tree and Tfae, due to the exponential dependency on /3 
for the probability to have such fluctuations, so between two consecutive 
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fluctuations the condition ^ still holdfU. Therefore, averaging ^ over 
all temporal fluctuations of Si during a large time window At — > oo in the 
steady state, one has 



7' 



XiFiSi ~ — (si), , (10) 

1 + 77' 

with (s,)t = liniAt^oo it E^lto'^*s^(^)■ Note that {s,)t = for T = 
and we recover so it is reasonable to assume that the approach (fTU]) 
holds for low (non-zero) temperatures. 

In order to compute the critical storage capacity let us consider this second 
scenario, namely the case of (3 very large and finite. In the limit of N, M ~f 00 
with a — M /N finite, one can assume the standard mean-field approach Si « 
(si), which is a good approximation for systems involving long-range interactions 
as in the case we are considering here, that is, a fully connected network. Under 
this assumption one has (si)t = (si) and the steady-state condition (jlOp allows 
to write to^(s) « = "Y^- £^[2 1^^^, {sj) — 1]. This quantity is related with 

the usual mean- field overlap function = J2j ^ji'^j) (where aj — 2s j — 1) 
by the expression 



r „u 7' 



to''- 1 ' B'', (11) 

1-1-77' V 1 + 77'/ 

where B^^ = J2j is typically of order O for random unbiased pat- 

terns. 

Expression (fTTj) can be used to calculate the steady state mean-field equa- 
tions for the system if one assumes that the system reaches a steady state in 
which the network has a macroscopic overlap with a particular pattern, the so 
called condensed pattern, with the remaining M ~ 1 being of order 0{1/\/N). In 
the following and without loss of generality, we choose = 1 as the condensed 
pattern. 

Using the probability H]) in the steady state it is easy to compute (s^) to 
obtain, for the mean-field overlap function. 



^^eftanh[2/3(/i,-0,)]- (12) 



N 

■I 

If we neglect the self-energy terms in ([7]) then 2{hi — Oi) ~ ^i^'^ ■ Inserting 
this into Eq. for /i = 1 and using pT|) . the steady state mean- field equation 
for m} = m reads 



m = ( ^tanh 



( — m + CI 

VI + 77' 7 



(13) 



where ((• • •)) indicates an average over a distribution V{C)- Here, C is a gaussian 
white noise due to the effect of M — 1 non-condensed patterns, and it is obtained. 



^Notc, for instance, that if neuron i is in the state s°° = 1 (one has hi > 9i) the probability 
to fluctuate to the state 1 — s°° = is 1 — p with p given by which gives (1 — p) ~ 
e-'2l3{hi-ei) ^ ]^ Similarly, if = (hi < Oi) the probability to fluctuate to the state 
s°° = 1 is p ~ e^''(''»~*») which is also exponentially small. 
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as we will explain later, taking the limit N oo in the term ^ 



7 



1+77 



-m for the condensed 



To derive Eq. 1131 we employed the expression m} 

pattern, after neglecting the 0{l/y/N) contribution in (fTTjl . Similarly and fol- 
lowing standard techniques, one can compute the spin-glass order parameter 
in our probabihstic approach, that is, q = ^ - tanh^ [2/3 (/i^ — 6'^)] (see for 
instance, ( Hertz et al., 199"l| )) which gives 



q = { ^tanh^ 



7' 



1 + 77' 



-m + C 



(14) 



and the pattern interference parameter r 
becomes 

r 



q 



i ^^_^-^(to'^)^, which in this limit 

(15) 



(i-/3tT^(1-'^)) 

Equations ()13m5p for m, q and r constitute the mean-field solution of the model. 
However, we must characterize the distribution 'P(C) to have a complete solution. 
From the calculations of Eqs. (|13m5p . we can obtain the expression for C, after 
taking the limit N ^ oo and self-averaging over the distribution of random 
unbiased patterns, from the variable 



(16) 



Considering Eq. pip . Q can be written as a combination of the variables 
^1 = ^J2^,=^i4^im'' and Z2 = -^T.t,^i<H^t B'' , and in the limit of inter- 
est explained above, both can be considered as uncorrelated gaussian random 
variables A^O, 1]. In fact, we have 

Q = ^J^—J- — -z^ - Va ( 1 - — 7 ) ^2 = Cizi - C2Z2. (17) 



1 -f- 77' 



For simplicity in the calculations, it is convenient to rewrite as C = Since 
zi, Z2 are normal-distributed, we can compute the probability distribution 7^(C) 



+ 



Ci 2 



p(zi)p(z2)c?zidz2 



employing standard techniques, that is, 7^(C) — J J ^ 
where p{z) is the normal distribution A^O, 1]. Computing this integral yields 
V{C) — A'^[0, cr^], that is, a gaussian distribution with zero mean and variance 
cr^ = 1 + Cf/C^ . This allows to consider 
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1 -I- 77' 



77 



7' 




1/2 



(18) 



where z is a normal-distributed variable N[0, 1]. 

Finally, the mean-field equations (after introducing the rescaled inverse of 



the temperature f3 



1-1-77 



7/3) take the form 



tanh 



P m + z\ 



■77 



7' 




(19) 



6 




(3 m + z\ ar + I 



1 + 77' — 7' " 



1-/3(1-9) 



(20) 



(21) 



The equations (|19ll2ip constitute the complete mean-field solution of the system 
for a working temperature near to zero 1 <C /3 < 00. It is noticeable that 
the effect of including synaptic depression competing with facilitation is not a 
simple rescaling of temperature (marked by the presence of /3) compared with 
the case of the classical static Hopfield model. On the contrary, the dynamics of 
the synapses affects in a different manner the signal and noise terms produced 
by the interference of the remaining (M — 1) patterns. This becomes evident 
in the explicit expression of the noise term (fTH| . Our results show that this 
term has a strong influence on the critical storage capacity when depression and 
facilitation are present, producing a non-trivial behaviour. 

Although Eqs. (|19ll2ip have been derived assuming 1 <C /3 < 00, one can 
give some arguments to extend their validity for any T if the system reaches a 
steady state (for instance, a recall, non-recall or a spin-glass state). In fact, for 
relatively high temperatures (and sufficiently low values of Tree and Tfae) the 
dynamics for both Xi (t) and Ui (t) is mainly driven for the fluctuating term 
which contains Si{t), instead of the deterministic exponential behaviour with 
time constants Tree and Tfae- Under this condition a plausible hypothesis is to 
consider both Xi (t) and Uj (t) as binary variables which follow the probabilistic 
dynamics of Si{t) and fluctuate in time between two possible values, namely x*-^' 
(u^^)) when Si{t) — 1, and x*^"^ (u'"^) when Si{t) = 0. A possible choice (but not 
the only one) for x^^'^'' (m*^^'"^) is the two steady state values of Xi{t) {ui{t)) at 
T = — see expressions in Eq. ([5]). This choice implies avoiding any temporal 
correlations or memory introduced by Tree or Tfae in the values of Xi{t) and 
Ui(t). Considering these assumptions one 



which gives again 



x.(0«l+(T+^-lj s,it) 

F^{t)^^^l + {J'-l)s,it), 

7' 

x,{t)F,{t)s,{t) = , s,{t) Vi. 

i + 77 



(22) 



(23) 



Note that in ([23|) there is now a dependency on t compared with (|9|). Now 
computing {si{t)) in the steady state using ([T]), as in the standard Hopfield 
model, one obtains again Eqs. (|19ll2ip which are, therefore, approximately valid 
for all the range of temperatures of interes10. However, this strongly relies on the 
assumption that a fixed point solution will be reached, and in general this may 
not be true for relatively large values of Tree and Tfae- In this situation, some 
stationary oscillatory states can emerge as a result of the presence of depression 



^Note that the effect of facihtation and/or depression in this approach is to change the size 
of the fluctuation between these two values for Xi{t) and Ui{t). For instance, for Tree, Tf^c — > 
one has x^^'l = x^*^) = 1 and u^^^ = = Use, so Xi{t) and Ui(t) do not fluctuate. 

^Sorne preliminary results in the limit of a — > have confirmed the validity of l lf 9l2f I I for 
any value of T < Tc , if the system reaches a stable fixed point (data not shown) . 
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and/or facilitation (sec, for instance, ( [Pantic et al., 2002} [Torres et al., 2007[ )). 
Concretely, the appearance of these oscillatory states is a consequence of the 
temporal correlations driven by the deterministic part of the dynamics (1310. 
This deterministic part, which is coupled with the stochastic fluctuations driven 
by Sj, can destabilize the fixed point steady states. As a consequence, the 
system starts to continuously jump between these metastable states. In this 
study, since we are interested in computing the maximum storage capacity, and 
this quantity is evaluated at T = 0, we will not find these oscillatory solutions 
and the mean-field theory remains valid. 

In general, Eqs. (|19ll2ip cannot be solved analytically. However, one can 
still get some information about the critical storage capacity because this is 
computed for /? — > oo (the zero temperature limit). In this situation one can 

perform the substitutions / ^= exp(— z^/2) tanh[/3(a2; + 6)] ~ ^'"^ (^iv^) ^^'^ 



J exp(-zV2) (l - tanh2[/3(az + 6)]) 



a/3 



exp 



( ^) 



I , which yield 



m ~ erf (y) 

1 [2 , 2, 
a/3 V TT 



where 



iV2 



2ar + 2a 



^ 1+77^' -7' ^ ■ 



1/2 



(24) 

(25) 
(26) 



Employing these approaches together with Eq. (pij) ones obtains a simplified 
expression for the complete solution, namely 



2a 1 




exp(-?/^) 



erf(2/), (27) 



where we assumed r ~ 1 in order to get a closed expression. This assumption 
works well as an approximation since r ~ 1 in the memory phase. Eq. (|27p al- 
lows to compute the maximal storage capacity for different synaptic conditions, 
as we will see below, including the competition between synaptic depression and 
facilitation. 



3 Results 

In order to obtain the critical storage capacity ac for a given set of values of 
the synaptic parameters, we have to find the maximal value of a for which 
nontrivial solutions (y ^ 0) appear. More specifically, we look for the max- 
imal value of a for which the stationary value of the macroscopic overlap is 
TO > 0.75. This criterion, which is usually taken for the numerical evaluation 
of the critical storage capacity in simulations, is illustrated for three different 
numerical examples in Fig. [TJ The figure shows that the value of the criti- 
cal storage capacity depends on the synaptic parameters, as it was found in 
( [Torres et al., 2002[ [Bibitchkov et al., 2002[ [Matsumoto et al., 2007| ). In these 
works, the inclusion of synaptic dynamics (in particular, synaptic depression) 
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Figure 1: Criterion chosen for the calculation of ac in Monte Carlo simulations. 
Each panel shows, for Use — 0.05,0.5,0.7 (from top to bottom), a numerical 
estimate of the critical storage capacity, obtained by averaging the stationary 
value of the macroscopic overlap (solid line) over many realizations of the stored 
patterns (dots). The value of Uc corresponds to the crossing point between the 
averaged overlap and the dashed line m — 0.75 that we used as criterion for good 
retrieval of the condensed pattern. Other synaptic parameters were Tree = 2 
and Tfae = 200. 



led to a monotonic decrease of the critical storage capacity of the network as 
one increases the synaptic parameters Tree and Use- This decrease was found 
to be caused by the loss of stability of the memory fixed points of the system, in 
the presence of depression. In Fig. (TJ however, we see that intermediate values 
of Use give higher values of ac, suggesting the possibility of a non- monotonic 
dependence of ac on Use, which is mainly due to the presence of facilitation. 

A more detailed analysis of this phenomenon is shown in Fig. [2lA. The figure 
shows that the inclusion of synaptic facilitation in a network with depressing 
synapses {Tree = 2, fixed) induces the appearance of a non-monotonic behaviour 
of ac with a maximum value for a given Use which depends on Tfae- The figure 
also shows the good agreement of our mean- field theory with simulation of a net- 
work of iV = 3000 neurons (symbols). In the particular case of r/ac = 0, we re- 
cover the results reported in ( [Torres et al., 2002| , that is, the fact that the static 
limit Ue ~ 0.138 is only obtained for t^ccUse = 0. However, if we include the 
possibility of synaptic facilitation in the synapses, one can obtain ae — 0.138 for 
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Figure 2: (A) Critical storage capacity ac as a function of Use for fixed r^ec = 2 
and different values of Tfac- The inclusion of facilitation causes the appearance of 
a non-monotonic behaviour of the critical capacity of the network as a function of 
VsE and t/qc, with a maximum which reaches the limit of static synapses (cxc — 
0.138). Different symbols correspond to numerical simulations of a network with 
iV = 3000 neurons and different synaptic parameters. (B) Behaviour of SNR, 
as defined in the text, for the same value of parameters as in panel A. This 
shows the origin of the non-monotonic behavior of the critical storage capacity 
found for different values of the synapse parameters. 



TrecUsE > 0, (that is, for synapses with a certain level of depression). This im- 
plies that dynamic synapses are not only convenient for dynamical processing of 
information in real neurons (Abbott et al., 1997 Abbott and Regehr, 2004), or 
to explain the appearance of global oscillations and other emergent phenomena 
in neural systems (Torre s et al., 2007| |Pantic et al., 2002} [Holcman and Tsodyks, 2006[ ) . 
In fact, an optimal balance between depression and facilitation is necessary to 
recover the high retrieval properties of networks with static synapses. 

We can also obtain a non-monotonic behaviour of Uc if we fix Use and vary 
the other synaptic parameters, as it is shown in Fig. [31 As a function of r,.ec, 
the critical storage capacity reaches the classical static limit for a certain non- 
trivial Tree value if facilitation is present. As a function of T^ae, the classical 
limit is also obtained providing that depression is not too strong (see, for in- 
stance, the curve for Tree = 10 in right panel of Fig. [31 where a large depression 
time constant induces lower ae values). The figure also shows the good agree- 
ment of mean- field curves and simulations (symbols). The appearance of these 
maxima in Figs. [IK and [31 can be explained due to the competition between 
depression and facilitation mechanisms. Once the system has arrived to a fixed 
point of the dynamics, the effect of depression and facilitation is mainly a mod- 
ification of the (fixed) strength of the synapses. Depression produces a decrease 
of the synaptic strength when the presynaptic neuron is active all the time. 
As a consequence, and compared with the static case, the pattern destabilizes 
for lower values of the noise produced by the interference with other patterns. 
This leads to a lower critical storage capacity value ( [Bibitchkov et al., 2002| 
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Figure 3: Left: ac as a function of Tree, for different values of T/ac- Facilitation 
induces the appearance of non-monotonic dependences in the critical storage 
capacity with a maximum which reaches the limit of static synapses. Right: Uc 
as a function of Tfac, for different Tree- For Tree relatively small, ae takes high 
values for the whole range of values of r/ac- The inset shows a detail near the 
maximum of the mean field curves. Other parameters used in simulations were 
Use = 0.2 and N = 3000. 



Torres et al., 2002). Facilitation, however, has the opposite effect as it increases 
the synaptic strength. Therefore, facilitation can enlarge the critical storage ca- 
pacity for a given level of noise with respect to the depressing case. The increase 
in the synaptic strength due to facilitation can only be induced until it reaches 
the static synapse strength limit — because the product XiFi in ([2|) cannot be 
larger than one. One can think that since these two mechanisms are regulated 
by different parameters, their competition would lead to the appearance of a 
maximum in ae- This argument is not sufhcient to explain the appearance of 
such a maximum since one has to consider that such competition affects both 
the signal term and the noise produced by the interference with other patterns. 
Only the consideration of the cooperative effect of all these mechanisms can 
explain the appearance of a maximum in ac, as it is observed. 

To measure how the relative strength of the signal compared with the noise 
is affected by the competition between these two synaptic mechanisms one can 
compute, for instance, the ratio between the signal and noise contributions to 
the overlap (see Eq. (fTn|) ). that is 

SNR = — (28) 

where we used r « 1 and m w 1 which is a good approximation at T = 0. 
For static synapses one has SNR = 1. One can now understand the maximum 
appearing in figure UK, for certain values of Tfac and Tree, as a function of 
Use, by inspection of figure [2j3. If one plots SNR as a function of Use one 
observes that it also has a maximum at a certain value Ug^, where SNR = 1, 
which is the value corresponding to the static synapse limit (in the figure this 
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corresponds, for instance, to Ug^ « 0.33 for Tree — 2, T/ac = 20). Therefore, it 
corresponds to the maximum observed in the behaviour of etc for the same value 
of the synaptic parameters (see figure For other values of r^ec, T/ac and 
Use, the shape of the SNR should be different but, similarly to the previous 
example, it can easily explain the non-monotonic behaviour of etc as a function 
of all these parameters. 

In general, we can see that large values of ac appear when Use and Tree 
have moderate values, and large enough r/ac- These values coincide qualita- 
tively well with those described in facilitating synapses of some cortical areas, 
where Use ^s. low compared with the corresponding values found in depressing 
synapses and t ree is several times lower than Tfac 

dMarkram et al., 1998D . To 
have such a relatively low value for Use is important because it allows for a 
stronger recovery of the synaptic strength due to facilitation. In addition, it 
is worthy to note that obtaining high ae values is possible for a wide range of 
synaptic conditions, as it is shown in right panel of Fig. [3] and more explicitly 
in FiglH For instance, high capacities {ue > 0.1) can be obtained for very dif- 
ferent values of Tfac- Since actual synapses usually present a high heterogeneity 
in the degree of depression Tree, and even more in the degree of facilitation Tfae 



(Markram et al., 1998 Wang et al., 2006), our results predict high values of a, 



for realistic conditions. This is shown in middle and bottom panels of Fig. [3] 
where the high critical storage capacity is mainly obtained for a wide range of 
Tfac (concretely, for Tfac > 10) around Use = 0.02, and Tree — 50. Assuming 
that our Monte Carlo time step is comparable with a typical refractory period 
of 2 ms, these values would correspond to Use = 0.02, Tree — 100 ms and 
Tfac > 20 ms, which are within the range of realistic values in several cor- 
tical areas ( [Markram et al., 1998D . Actual neural systems could, indeed, take 
advantage of this peculiarity to preserve a possible fine tuning of the degree 
of facilitation for other purposes, such as a fast dynamical processing of data, 
while an optimal recall of the memories is conserved. 

The improvement in the critical storage capacity for these realistic values 
(Use = 0.02, Tree = 100ms and Tfae > 20 ms) in comparison with the case 
of only depressing synapses is highly significant. Looking at bottom panels of 
Fig. HI for instance, one can see that the critical storage capacity reaches ac — 
0.138 for the parameter values mentioned above. However, if we consider only 
the effect due to depressing synapses (that is, we set Tfac = 0), we obtain ac — 
0.07. That is, the capacity decreases around 50% of its value with facilitation. 
This indicates that facilitation could have a highly important role in the storage 
and recall of memories. 

As an illustrative example of the implications of the results reported above, 
let us consider a system constituted by a network of N fully connected neurons 
which receive an additional weak external input during a very short period of 
time. The total synaptic current to each neuron i then becomes hi{s, t) + hf^*^ (t) . 
If the system is in the memory phase, we expect the external input to drive the 
system towards a stationary attractor state {■ff"}. We will also consider that 
the input stimulus occurs periodically in time as follows 

f^extf^^ f tn<t<{tn+6t) \/n = 0, . . . , Ninputs ^^g) 

^ ^ ' \ otherwise ^ ' 

where /i <C 1 is the amplitude of the weak input, i„ is the time at which the n— th 
input event occurs, 5t is the duration of a single input event and T = — tn 
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Figure 4: Left: Surface plots of the critical storage capacity ac as a function of 
different synaptic parameters, for fixed Tree = 2 (top), Tree = 50 (middle) and 
Use — 0.02 (bottom). Right: Contour plots which correspond to the surfaces 
of the left. Regions inside the lines corresponds to a set of parameters for which 
the critical storage capacity is high. Middle and bottom panels correspond to 
a more realistic set of parameters (if one assumes that the Monte Carlo step is 
of order of a typical refractory period of 2 ms), and they illustrate that high 
capacities are obtained for a realistic values of the synaptic parameters. As one 
can see, the inclusion of facilitation is able to double the critical storage capacity 
for certain situations (see main text). 
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Figure 5: Memory retrieval under external stimulus for a network of = 400 
neurons with dynamic synapses. The network is stimulated with a periodic 
weak input train (middle panel) which, in the memory phase, is able to induce 
the retrieval of a certain activity pattern (constructed in a band-like fashion to 
allow a clearer visualization). The value oi Use is slowly incremented in time 
(bottom panel) while the synaptic parameters Tree — 2 and Tfac — 20 remain 
fixed. Good retrieval occurs only for a certain window of values of Use, which 
shows a non-monotonic dependency of adUsE)- 



is the period of the stimulus. If the system is in the memory phase, the stimulus 
will lead the system into the /xp — th attractor and the memory retrieval will be 
successful. Otherwise, for large number of stored patterns, namely M > acN 
the system will fall in a spin-glass state characterized by a mixture of a high 
number of patterns, and the retrieval will have failed. We explored how certain 
synapse parameters affects the retrieval process under this type of stimulus. 
This is shown in Fig. [5] for a network of = 400 neurons and M = 48 patterns 
(a = 0.12), with the pattern no constituted by consecutive groups — 100 neurons 
each — of alternate firing and silent neurons, and the remaining M — 1 being 
random unbiased patterns. As an example, we consider dynamic synapses with 
fixed characteristic time constants Tree = 2 and r/ac = 20, and Use varying in 
time for the whole duration of the stimulus. The figure shows that facilitation, 
which induces the appearance of a non-monotonic relation aeiUsE), allows for 
a good response to the external weak stimulus for a certain window of values 
of Use- In particular, for values of Use — 0.4 the stimulus is able to drive the 
system towards the attractor and recover the corresponding memory pattern /iQ. 
The range of values of Use at which the system retrieves the pattern coincides 
with those between the points at which the line a = 0.12 crosses the critical 
mean-field line ae{UsE) showed in Fig. [2] for Tree = 2 and Tfae = 20. A similar 
type of behaviour also occurs fixing Use and Tree and varying now Tfae (data 
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0.05 0.08 0.11 

Figure 6: Basins of attraction of a neural network in the presence of dynamic 
synapses. We can see that facihtation enlarges the basins of attraction with 
respect to synapses with only depressing mechanisms. This allows to retrieve 
the previously stored patterns even if the initial condition for the network is 
only weakly correlated with the corresponding pattern. Parameter values are 
Use = 0.2 and Tree = 5. 



not shown), which also shows the main role of facilitation in memory recall. 

The effect caused by dynamic synapses does not only affect the stability of 
fixed points of the system, which gives us the critical storage capacity, but also 
the dynamics of the network. This has been recently investigated for the case 
of a single stored pattern {a 0) at finite temperature ( [Torres et al., 2007| . 
For many patterns, however, the interference among them can influence the 
behaviour of the system near the attractors due to the appearance of many 
local minima associated to spin-glass states ( jAmit et al., 1987a[ ). To analyse 
this effect, we can define the basin of attraction of an activity pattern fj, as the 
minimal value for the initial condition, m^, that allows the system to tend to 
this attractor ( Bibitchkov et al., 2002[ Matsumoto et al., 2007). The measure 



of the basins of attraction is highly relevant because the system will not easily 
tend to the stored attractors if their basins are too shallow. Since we know that 
dynamic synapses have indeed a notable effect in the dynamics of the network at 
finite /3, we expect them to infiuence the dynamics also at f3 —^ oo, and therefore 
modify the basins of attraction in some way. We have explored this issue, and 
the results are shown in Fig. We can see that the inclusion of facilitation 
in a network of depressing synapses leads to an increment of the basins of 
attraction. Since basins of attraction are associated with the error-correcting 
ability of the systend, our results show that networks with facilitating synapses 
are adequate for recovering patterns, even from initial conditions only weakly 



^ The error-correcting ability resembles the capacity of an attractor neural network to prop- 
erly recognize an activity pattern when the given initial condition is not strongly correlated 
with the corresponding pattern. 
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correlated with the patterns. One can conclude, therefore, that facilitation 
increases the stability of the fixed points by increasing the basins, and this 
leads to a higher critical storage capacity, as we have reported previously. 



4 Discussion 

In this work we have focused on the role of the competition between several 
synaptic activity-dependent mechanisms, such as short-term depression and fa- 
cilitation, in the capacity of attractor neural networks to store and retrieve infor- 
mation codified as activity patterns. Previous studies have found that depress- 
ing synapses drastically reduce the capacity of the network to properly retrieve 



patterns (Bibitchkov et al., 2002 Torres et al., 2002 Matsumoto et al., 2007). 



These results highlight the role of depression on the processing of spatio-temporal 
information at short time scales (which allows for the appearance of dynamical 
memories), in detriment of its function in stable recall necessary for memory- 
oriented tasks. The consideration of additional potentiating mechanisms, such 
as synaptic facilitation, turns out to be convenient then for memory recall in 
these dynamical conditions, reaching in some cases the static limit Uc — 0.138. 
This leads to think that synaptic facilitation could have a crucial role in the 
performance of memory retrieval tasks, while maintaining the well known non- 
linear properties of dynamic synapses, convenient for information processing 
and coding ( [Abbott and Regehr, 2004[ ). 

Our results also indicate that the range of parameters for which facilitation 
allows to have a good memory performance is notably wide, and therefore, 
these benefits can be achieved without a precise fine tuning of the synaptic 
parameters of the model. For instance, it is well known that dynamic synapses, 
and in particular facilitating synapses, usually present a high heterogeneity in 
their concrete characteristics (Markram et al., 1998 Wang et al., 2006] ). Since 



the conditions for which we found high critical storage capacities for random 
unbiased patterns (i.e, > 0.1) are very general, in the framework of our model, 
this can support the idea that actual neural systems could indeed take advantage 
of this fact to perform additional tasks — which are considerably different from a 
dynamical point of view — while the optimal access to memories is maintained. 

Although we have derived a mean-field theory for unbiased random patterns 
/ — 0.5, there exist other mean-field approaches in the literature which can cor- 
roborate our main conclusions about storage capacity and can be useful to ex- 
tend our study for other types of stored patterns. One can employ, for instance, 
the mean- field theory developed in ([Shiino and Fukai, 1993 ) valid also for other 



values of /, or the one presented in ( [Tsodyks and Feigelman, 1988[ ). Our ap- 



proximate theory presents, however, several differences which we consider of 
convenience here. Concretely, the fact that it allows to work with a network with 
temperature, even in an approximate way, represents a significant practical ad- 
vantage. It could, in principle, be a good approximation for high temperatures, 
and preliminary results confirm this hypothesis ( [Mejias and Torres, 2009[ ). In 
addition, the assumption of having the same threshold level for all neurons, as is 
done in ( [Tsodyks and Feigelman, 1988 ), seems to be too restrictive for the mod- 



elling of biologically motivated neural networks due to the well known variability 
observed in the voltage threshold of actual neurons ( [Azouz and Gray, 2000[ ). In 
our work, however, this experimental fact is taken into account by considering 
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a threshold 9i given by ([5]) for each neuron and which induces the existence of 
the noise term (which, as we have seen, has a strong effect in the behaviour 

of ttc.) 

In order to treat the effect of dynamic synapses, and concretely of short- 
term depression and facilitation, we have employed a simple model for synapse 
dynamics ( [Tsodyks et al., 1998[ ). The predictions of this model agree with the 
experimental data from cortical slices, as one can see in (Markram et al., 1998) 
(see explanation in the results section). However, there are more realistic mod- 
els which could be used to test our results. It is known, for example, that 
the stochastic nature of the transmitter release could play an important role in 
synaptic fluctuations ( [Dobrunz and Stevens, ig gT*). Models which take into ac- 
count this stochasticity (such as ( de la Rocha and Parga, 2005) ) could be used 
to test our results with fluctuating synapses, although the complexity of such 
stochastic models would not allow to develop a simple mean-field theory, even 
approximate. 

It is also known that dynamic thresholds are responsible for several complex 
phenomena in ANN ( [Horn and Usher, 1989 ), that could be similar to the ones 
observed in ANN with dynamic synapses ( Pantic et al., 2002} [Torres et al., 2007 ). 
This could lead us to think about the influence of dynamical thresholds in the 
network critical storage capacity, and its relation with the results presented 
here. While this is an interesting issue not reported yet, it is worth not- 
ing that although dynamical thresholds also induce the appearance of oscil- 
latory states similar to the case of dynamical synapses, a direct mathemat- 
ical relation between the dynamics of thresholds, as the model reported in 
( [Horn and Usher, 1989 1 and the phenomenological model of dynamical synapses 
by ( Tsodyks et al., 1998 ) cannot be derived (see discussion about this important 
issue m dPantic et al., 2002D ). 

Finally, and attending to the dynamics and error-correcting abilities, the 
effect of synaptic depression on the basins of attraction has been previously 
studied ( |Bibitchkov et al., 2002[ pVIatsumoto et al., 2007 ). In these works, neu- 
ral networks with a general inhibitory contribution are considered, and sev- 
eral assumptions such as a fixed threshold value for all neurons are made. 
On the other hand, our study considers general networks in which excitation 
and inhibition are treated in the same way (in particular, our neurons are not 
purely excitatory or inhibitory), and each neuron possesses its own particu- 
lar threshold value which is also in agreement with several experimental ev- 
idences ( [Azouz and Gray, 2000D . Our study shows that facilitation enlarges 
the basins of attraction compared with the case of only depressing synapses. 
As a consequence, we find that a convenient balance between synaptic de- 
pression and facilitation is necessary for neural networks to work optimally 
at different dynamical tasks. This is in agreement with recent experimental 
results which show a heterogeneous level of depression and facilitation in real 
synapses dMarkram et al., 1998||Wang et al., 2006D . 
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